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ABSTRACT 

Educational practitioners tend to resist or avoid 
doing educational research primarily for two reasons: their prior 
experience with statistics, computers, and research design has 
produced insecurity and anxiety; and, they have acquired a basic 
misunderstanding of the nature and purpose of research. Good research 
should be both technically sound and practically oriented. 
Practitioners can develop researchable questions from their own needs 
and from elements in their own environment: a special population with 
which they deal, a procedure they are considering trying, or a 
problem that they want to solve. There is a common misunderstanding 
that real research must utilize a true experimental design; however, 
a qualitative research design may be more appropriate for educational 
settings. Examples of these designs include: comparing two or more 
groups with regard to some outcomes; evaluation research or impact 
assessment; correlational research; descriptive studies; and case 
studies. Words, not numbers, may be a more appropriate focus when 
collecting data. The data can be summarized using the matrix or table 
shell method. In addition, using multimethod research designs which 
combine quantitative and qualitative approaches to collect and 
analyze data can increase the internal validity or bel ievabi 1 ity of 
findings and conclusions. When it is necessary or appropriate to 
utilize quantitative procedures, the research project will proceed 
more smoothly if the educational practitioner follows certain 
guidelines in working with statistics and computer specialists. 
(lAH) 
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Introduction 



Research. The ver>' term often strikes fear into the hearts of graduate students. 
Images come to mind of piles of printouts and unintelligible statistical jar2on. Above all. 
research is seen only as a means to some very limited ends. These include completinc a 
course requirement, master's thesis, doctoral dissertation, or a publication written soielv 
for promotion and tenure purposes. Terror and tedium are often synonymous w ith the idea 
of research, while enjoyment is not. 

These images are unfortunate ... and. above all, unnecessary. All of us. whether 
graduate students, classroom teachers, or administrators, have what it takes to do 
•meanmgful" research and at the same time bypass a great deal of the pain. 

Much of this fear and avoidance is associated with "the numbers part" of research. 
Lefs face it: computer programming, statistics, and research design are not particularlv 
high on even-one's "favorite course" list, hutially negative experiences in these classes are 
partly to blame for a spillover effect which translates into avoidance of research activities 
outside of school requirements. 

It shouldn't have to be that way. Fact is. plenty of outside help is available ... 
A.N/D accessible ... in the technical areas of statistics and computer use. Of course, vou 
have to know how to look for it, as well as how to use it most effectively once you do find 
it. That IS the primary purpose of this paper. 

But strange as it may sound to you, all of those impressive-looking tables, oraphs 
and numbers are NOT the heart and soul of research. In the honest judgment of this 
statistician. YOU are actually the driving force behind good, solid, researchable probleni. 
That IS because of the richness of daily experience which you. as educational practitioner, 
encounter ever>' single day. In all honesty, numbers for the sake of numbers aren't too 
meaningful. However, take those very numbers and apply them to an interesting 
classroom problem - which is how I get my greatest satisfaction in workino with vou 
and they take on a life of their own. Simply put: YOUR great ideas, needs^to knou and 
outnoht cunosities are where "good research" originates. It's hard to imagine a teacher or 
administrator who doesn't, from time to time, wonder "why." "how" or "what if" as he'she 
goes about the daily business of educating students. Take one of those questions, discu.. 
.t vv .th a technician like me who can help you go about the steps of answerino it and 
you ^■e got yourself a research project. Furthermorcit is one that will be more than a .rade 
tor a course: the answer to your question will obviously mean something "real" to you^ 

This paper is an invitation to you to accept that challenge. It beains with an honest 
look at the reasons people typically avoid research: the fears, myths and other causes of 
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resistance. These banners wiil be followed by a ver>' basic outline of what "real" research 
is intended to do. This will include a look at the "elusive ideal" of the true experimental 
design - the one you probably learned as -best" in your required research methods course - 
- and how, surprisingly, this method may fall far short of the mark for what vou as an 
educational researcher need to find out. Sometimes the best way to study what you need to 
study is to gather your data in words: qualitative research designs and data collection and 
analysis procedures. In fact, the "wave of the future" is to combine both numbers and 
words, or multimethod designs. Next, some tips will be presented on how you can 
approach the research process with far greater confidence and make the best use of the 
technical help that you may need. 



Traditional Binriers to Doing Researrh 

As mentioned above, people tend to shy away from research, particularly if it is not 
a requirement for a course, a degree, or a condition of promotion. The reasons for such 
avoidance stem primarily from two sources. The first of these is an unfortunate spillover 
effect from prior coursework experiences in statistics, computers and research design. The 
second has to do with a basic misunderstanding of the nature and purpose of research. 
Some typical barriers are identified below. 

^iiiloo hard (corollarv- "You h.ve to be a aenius"y" This overall indictment of 
research activities is primarily a combination of both general barriers discussed 
above. Images of Einstein come to mind and generate an unfair comparison to 
oneself and one's assumed lack of qualification to conduct research. Quite the 
contrary - anyone with a need to know anything has the potential makings of a 
"good, researchable" question. That is why, in my estimation, educational 
practitioners such as teachers and administrators are actually in the BEST position 
to idemify a solid, practical, and above all useful research question - gencrall> 
emanating from their own day-to-day experience. 

l it takes too l onoj Horror vudies told by lifetime doctoral A.B.D.s are to blame 
for this one, as are recollections of Margaret .Mead-type decades-long immersions i 
the field being studied. In fact, some of the best and most useful studies are 
very limited in term.s of time and cost considerations. 

"''"^ no statisticia n." ( o r "c o mputer expert." •■rp<;P.nrrh>>r^ o^ Granted, "the 
numbers part" has traditionally also been the 1-LARDEST part. However, there is 
no law that says one researcher must do it ALL. is there?! Collaborations between 
technical specialists (like me) and content-area practitioners ( like YOU!) aenerallv 
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lead to the best "meeting of the minds" in terms of shared expertise. For another 
thing, it may surprise some of you to discover that numbers are not the onlv. or 
fcVEN tlie BEST way, to collect and analyze your research data! More on that point 



later 



JilLonU' for college professors ror mnsterVtheses^ordoct or^ dissertations).- As 
mdicated earlier in Point # I, it's the rare teacher or administrator who hasn't faced 
a p.ractical problem or need to know something in the course of doing his/her dailv 
job. Is it a student with a panicular leaining disability? A challense to fmd a wav to 
motivate one's teaching staff during tough times when monetary- rewards 'are 
unlikely? A desire to try a totally different approach to teaching mathematics in the 
junior hjgh grades? A frustration with ones inability to stem the tide of hiah school 
dropouts? All of the above - and MORE - have tremendous potential for turning 
into research studies which are technically correct, doable ... and above all. 
USEFUL in answering a need co know. 



The Essence of Research 

ft may come as a surprise to you that "good" research can (and SHOULD) be both 
techn.cal|> sound AND practically oriented. With terror in our hearts, we tvpicallv recall 
the unwieldy, 30-step statistical formulas and accompanying multisvllabic jaroon . and 
thus think 01 research only in the first regard. Yet the true, underivino purpose behind 
even the most sophisticated quantitative treatment is to answer an actual question ... and no 
more. To put it another way, without some "real-lit-e substance" behind it. number- 
crunching for its own sake is absolutely worthless. 

Thus, in actuality, the whole process should begin with an idea, curiositv or need 
to know - it's that simple. A "good statistician" can. and should, take it from there 
Computers and statistics are ver>' much the supponing players in this scerrano ... no matter 
hru elaborately costumed they may bel No doubt about it: the real star of the show ,s the 
research question. 

As mentioned previously, you as educational practitioners have a wondertullv nch 
and close-up view of the ver>- raw matenal of such research questions. It would be hard 
tor me to imagine a day in which you go about the course of your work and don t 
onder ' about somethino. 

All practitioners essentially face these "3 P's' : they deal with people, apph 
procedures, and identify and solve problems. Researchable questions could be develooed 
trom any, or all. of the foilowina elements: 
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A special emim to del «i,h ,e.g„ smden.s wi.h a particular leanrino difficulrv 

or. ,n .he caseof .if.ed students, a challenge: firsr-year teachers: staff mentbers': 
parcnfs) 

A a. to tr>. (would a new teaching method work better? how about a new 
Tsuhs';," " " " '"^''"^ -'""^ '° SC. better 

A eroblem to be solved (why isn't X working?) 

■^■I^^^JlS^Em m^n,' An He.an, (but Often lr,.l.,.,nt) Me aLfa^eseairh^^ 
it's not atri"" ^^ocational research is "if 

ha fT , '° - — -fely assume, -1 got 

on I n 1: " cause that crept in and 

a„ r, ■ "T" ""'""'^ -^J-- °f -e's studv, and then 

e s h" a d '° ■*"""™'' ^''f^^'^ tested (e.g.. who 

"she he, groups. Thus, 

h he helps to get a more focused look at the actual effect of the 'one key difference' that ,s 

>-iid„; or confidence ,n the results is enhanced (Borg and Gall 1989) 
idea, oftlr","', P^^^""'™^-™ «-^^Wy relate to the rather elusive 
I t etai oo r M ' ■ ""'^ ^>-^-™^- ^replanned-down-to-the- 

cm e r ,T " ""^'P"''- '""""^ ™ ^ -e hardlv 

"rr V :t ; '-^ -ver. 

de"o»n c " ~- Percepttons. 

X ,h "T"t" -^'^^ =™P'>^ — ^= rolled 

ua ta e rl H '"^ ^^"'^'^^^ '^bis is where 

qua„ta ,>e esearch procedures - that is. using words rnstead of numbers to »ct at 

rounded theor, " - are panicular.y valuable. .More will be said on this pent late'r , In 
l-rar„„c,a , v. foreducatronal researchers. Yet. unfortunately, the mvhical ideal of the 
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bnsic i„„oduc,ory research classes .akon by educators in ™as,er's degree proc„„,s and .he 

rll , ca,r °' -"-h stuCes^ In ,.„re 

rca . cally encountered) educational settings which lack such absolute control and abilitv 

01 0" n?'"' ' "r"" °' ^"'"■-■-^ --'=^1- Th^ 

989 1" '° " • P^"'"' 'M-shal, and Rossn,a„ 

est 7 ' "^>-°-f°"°- --hing research needs or 

q-sttons to a vanety of alternative research designs using qualitative data collection 

d Marl" " ^"^^""^ ^"""^ 

pmc ,ca real, es of do.ng educational research with "real" subjects and in "real" settings, 
r' s n!|v"h ■ 7'T ""^"°°'^-'>-P=- °f =^"--al research with" a 

,n r„duet 7 " ™' "'^"^^'■^^ -^>' '-^P- '° ™— 

in conducting educational research.) 

inciudef:rj::r:::ir^" °^ --'--^'^ 

pfflBann^ or tnore .roups with te fflnijo^oa^aitcome te.g.. do sixth-rade 
bo>. and g,rls differ significantly with respect to average science aptitude sc^ores^ 
h.gh school freshmen, sophomcres.juniors and seniors differ sisnificantiv with 
regard to attitude towards the quality of extracumcular activities ;ffered bl their 
school . ). Thts comparative-type of research scenario stans with a curiositv' about 
-h>- some outcome seems to van' and tries to explain' or 'account' for it bv 
Pomttng to some grouping condition as the primary reason. (For instance, the 
outcome of tnterest might be the science aptitude scores. Bv "pile-sonin»- into two 
groups - boys and girls - the researcher might discover that the vacation or 
d.lferctce. in the a>.erage science aptitude score ,s in elTec. "explained" bv lender 
He or she may find that such grouping by gender results in "»ithi„-oroup"''scores 
be.ng more alike" and "between-group" scores being "more differen"," ,e » bovs' 
scores vary or differ less among themselves than do the average scores be'tueen 
boys and gtris,. What you will be doing, when you "test the sianificanee" of such a 
aroup dtlference, is seeing if that average difference that voj've observed is "bi» 
enough" to safely assume that these groups would indeed differ on average - or if 
..could have been due to chance .e.g., more a refiection of your panicula^r sample 
.b.^n a true dtfference, (Jaeger, 1990), For our science e.xample. suppose that we 



nave observed a 3.5..point average score difference between the sixth-cradc bovs 
and g.rls ,n our study. In a scientific research study, we will essentially be askina'if 
such a 3.:. difference is "big enough" to assume with confidence that sixth-^rade 
boys and girls will indeed differ in general. On the other hand, we mav find that the 
test statistic indicates that 3.5 points of difference could have occurred solelv due ta 
the vagaries of OUR PARTICULAR individual san^pie - and over the lon'a ranoe 
(typ.cal sixth-grade boys and girls that would be considered the "population" ^rom 
whom we drew our "sample" subjects for study) would not, on average, differ). 

It is important to note that you can do such a between-group comparison 
^ THOUT having had an "experimental-type" treati..nt (e.g., a special tutonal or 
lesson g,ven to one group) under your direct control. This would make your 
des.gn an "after-the-fact," or ex^ostfacto (also called CMsaUmnEaratlye) research 
design (Borg and Gall, 1989). Of course, you have to be much more careful in this 
case to make sure that you've identified, and ruled out, as many other possible 
causes" of your difference. For instance, in the science example, we want to be 
reasonably sure that "gender," or the grouping variable that we have selected is 
•ndeed the "cause" of the average between-group science score difference. Without 
the benefits of the tight control and randomization (e.g., "getting a good mix" on all 
other possible causes across boys and girls), it is entirely possible u.at some other 
variable (e.g., differences in aptitude or intelligence) actually "caused" the 
d.fference that we are accidentally attnbuting to gender. The researcher can in 
essence do some solid brainstorming and detective work, to identify as many si-h 
outside causal factors as possible, and check for whether bovs and aiHs' were 
actually different on them. If he/she can show that these factors were abolit equallv 
distributed between the groupings of boys and giris, then the researcher can more 
confidently rule out these outside explanatory influences and safelv stick with 
gender as the "cause." Another procedure which may be available with 'the help of a 
statistician is to "mathematically slice out" any such inherent outside differences 
between the groups so that the researcher amves at a "more punfied" measure of 
the group difference (e.g., gender) which was onginally of interest to him or her 
This quantitative procedure of "correcting, after the fact" for such factors which the 
researcher did not have the "luxury" of randomly mixing between/amono the 
groups beforehand, is known as covariance annlv.i. n.^g.. 1990) 
yiiaiLa_g articular educntional poli c v . program or p rn...,,". .,,„,f,,d" in 
obtmmn^h_^des^ For instance, did a special ev^;;;;;; series of job- 
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hunting-tips workshops increase the number of job interviews and job offers of 
high-school dropouts in a given community? This is known as evaluation re.ccnrch 
or lr n . pnct assessment . As pointed out by Dereshiwsky and Packard ( 1992). 
evaluation research seems to be the most universal and applicable branch of 
scientific inquirN'. Most, if not all. of educational eno.avors involve some son of 
".nter.'ention" (a new method of teaching: a counseling program: a tutorial scnes) 
targeted at a predefined set of results which the "intervention" is designed to 
accomplish (increased academic achievement; improved self-esteem: other academic 
social, emotional and behavioral outcomes). Lots of readable references exist 
dealing with how to plan and implement evaluation research studies (Patton. 1990: 
Berk and Rossi. 1990; Rossi and Freeman. 1989). 

Seeinaj >-hat cpes with what " Identifying the direction and amount of change in 
some outcome (e.g., increased self-confidence) as a result of a change in direction 
and amount of some other outcome (e.g., improved academic achiev-ement) would 
constitute a corrdajional research design. In a way. this type of research question is 
the polar opposite of the preceding one: rather than focusing on WHY THERE ARE 
DI FFERENCF.S . the practitioner wants to know WHAT'S RELATED OR 
AS SOCIATED . Knowing how, and how much, one outcome changes with a 
change in some other outcome could be very useful in a predictive sense.^especiallv 
if one precedes the other. Again, the researcher needs to exercise particular care 
before assuming that, just because two things 'move together.' that one necessarN- 
"caused" the other. For instance, there could have been a third, "hidden" cause of 
BOTH the increased self-confidence AND the improved academic achievement, 
such as a special one-on-one tutoring experience with peer counselors that 
happened to run concurrently during the study time. Once again, however, the 
astute educational researcher will "play detective" and either rule out or 
mathematically remove as many such effects as possible. This will be done to "zero 
in" more efficiently on the two (or more) factors that he.'she has preidentified and 
wishes to understand in a correlational or relationship sense. 

'llmJMckin^ar^^ Yes - this one is actually OK to do! In fact, it mav be 
critically needed. Packard (1992) has warned of the danger of going ahead'and 
blindly applying textbook models of some educational process or procedure - 
uhich may look slick on paper but may also have little to do with life as it is lived in 
actual classrooms by students, teachers and administrators. 




The ideal way to "emerge" such models is through "grounded theory" approaches: 
idenLifvLnji, in a descri£tive study design sense, the key issues, problems, ond 
factors faced by these important players on the stage of an actual school setting, 
This in turn may need to involve extensive use of qualitative procedures, or having 
the words augment (or even replace) the numbers. Only after sifting through these 
"real-world" data can the researcher validly begin to piece together a "model" or 
process of what he/she thinks can account for, or explain, what is actually 
happening in the school setting. 

Naturally, such a model can then be "road-tested" in other, similar situations or 
settings to see how well it holds up, or if it needs to be modified in order to be more 
generally.applicable, Yin (1989) has referred to this step-by-step validation and 
rer.nement as "cross-case analysis" (p. 57). He and other quah'tative researchers 
such as Miles and Huberman (1984) have advocated selecting "extreme" cases, or 
polar opposites - e.g., schools identified as having "e.xceptional" and "suboptimal" 
climate by whatever means of classification - and seeing if the model "holds up" 
for the condition that it is expected to (e.g., are the factors in the model "present" 
for the "good climate" schools and "absent" (or at low levels) for the "poor climate" 
schools?). Yin has termed this process "literal" vs. "theoretical replication." 
respectively (p. 53). 

In summary, the educational researcher may simply not "know enough yet" about 
the underlying process, procedure or model to be ready to jump to the next step of 
testing similarities and differences. In other words, according to Packard (199:1. 
he/she first needs to know "what it is." This would necessitate the son of 
descriptive research study described above, as an essential prelude to refinement 
and (eventually, upon gradual development of a well-validated and reasonabi> 
generalizable model) the more traditional expenmental research designs, in order :o 
test for causes, effects, differences, and the like. 

In the r -eceding alternatives to true expenmental designs, a number of references 
have been made by this dyed-in-the-wool statistician to the use of qualitative research 
procedures. This may seem surprising, given the "feared/revered" phenomenon associotino 
u uh statistics. On the contrary, collecting and analyzing data in words, as opposed to (or in 
addition to) numbers often makes for a tighter, more believable studv. Thus, before 
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proceeding to a discussion of numbers and statistical help per se, a brief overvieu of the 
nature and role of qualitative and multiinethod research procedures is in order. 

The Power of Words; Rich nnH Rpv^njin^ 

As mentioned in Banker # 3, pgs. 2-3, nutnbers may not necessarily even be the best tool 
for you to use to address your research question. Ironically, for all of my love of thinas 
quantnat.ve, since coming to Northern Arizona Urnversity IVe spent the greater pan of rnv 
t.me on qualitative data collection and analysis techniques. The resea.-ch world is on^duallv 
commg to the (long-overdue) realization that words are superior to nun.ber. in a great deal 
ot context-nch research situations such as those that practitioners face. Interviews lo^, 
books, wntten diaries, both official and unofficial documents - these sources, and more", 
may be just what you need to answer your research question - instead of the computer 
programs and statistical tables. 

Vin (1989) has nicely elaborated on the different (not "better" or "worse") settinos 
m which one might wish to use words vs. numbers. He refers to this distinction as 
analvtic vs. "statistical generalization," respectively. 

The external validity- problem has been a major bamerin doina case studies 

So : . f ^uc ^^^f,^^ ^^^^ ^'>S'f. - basis for^g^neralizin : 

research wl^r. ^[^'^IPl'^'^'y ^contrasting the situation to surveV 

research, where a sample (if selected correctly) readily Generalizes tn \ 
larger universe. This analogy to samples and universe IfncoTeT^^^^^^ 
dealing mth case studies. This is because survey research rel e on 
■^'cmt,cal generalization, whereas case studies (as with^experiment d^^^ on 
unalync generalization (p. 43, emphases in original text). ' 

Yin goes on to describe analytic generalization as wishing to obtain an in-depth 
understanding of a locally based process, procedure or phenomenon. This would involve 
looking at multiple facets or charactenstics of the entity being studied, such as for instance 
a Mngle classroom, student population or teaching procedure. Such analvtic aeneralization 
contrasts with the more conventional statistical generalization, the latter involvina a desired 
projection ol a more greatly limited qtiantity far beyond the scope of ones studv settino 
An example of this would be "I predict, with 95% confidence, that the'averacTe 
(Standardized Test name) score fc- fourth-graders generally will be between 87 Sand 91^^. 
points. If It ,s indeed yourdesire to make such broad generalizations of limited quantities 
across thousands or even millions of subjects, than you definitelv would need to emplov 
mforent.al statistical procedures to test your assertions. However, if you are lookino to 
understand a wider variety of factors with regard to YOUR school system or locallv unique 
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populations of students and teachers, you may need to "go qualitative." For one thing, 
some ,f not all of these factors may not be readily "numherizable." For another, the 
richness of the words as spoken or written by these subjects may give vou a more 
inlormative and revealing look at what is really happening, 

Qualitative research has truly come info its own element as a lono-overdue. and 
uni.I recently misunderstood, branch of scientific scholarly inquir7. A number of sources 
(Patton (1990); Marshall and Rossman (1989); Krueger ( 1989); and Denzin (1978), to 
name but a handful) have documented how to apply such procedures to educational 
research settings. As stated by Dereshiwsky and Packard ( 1992): 

Learning to collect and analyze qualitative data requires relatively little 
raining, as compared with inferential statistical procedures (and 'often 
rela ed computer programming skill). Yet for all its surface simplicity a 
quotation can be tremendously revealing in terms of its wordino content 
time'"^h?r; l""^ ^'-K'^' ethnographers have known forq^uite some 

words MnV ^"'^j'^'^^^ ^ ^^°^y in their own 

, Jm the evaluation researcher gains a valuable peek into the 

li ve°d a d°L^ ^ '"^'T^''-' ^"j' i-P-^ °f ^he program IsTelS^ 

lived and expenenced it from a unique and personal perspective, (p, 4) 

Easy-to-use procedures also exist for summarizing these words into conveniently 
readable and understood form. The most popular of these is a chart or graphic known as 
the -matn-x" or "table shell" method, developed and creatively illustrated bv Miles and 
Huberman (1984). Summarized key themes appear in the "boxes" of this graphic, enablino 
the reader to spot key themes, trends and between-group comparisons at a slance. Packard 
and Dereshiwsky have extensively applied the matrix method in a number of evaluation 
studies. These include a baseline needs assessment of the organizational climate of a private 
high school located in a Navajo reservation community (1991, November). A second 
recent application involved pilot-testing of educational materials for use with Navajo 
families identified as being "at risk" for developing diabetes ( 1991 . August). The reader is 
referred to these publications for illustration., of the matrix method of compilina qualitative 
data. ,A panial listing of recent qualitative evaluation research publications bv Packard and 
Dereshiwsky appeals as Section B of the references to this paper.) 

The second method of compiling qualitative data is known as the "summary 
narrative" method (Denzin. 1989; McCracken. 1988). It consists of writing a creatlv 
condensed summary of key themes, trends and other information gleaned from re^adino 
reviewing and coding the totality of qualitative data. Such summary' narrative is the^n 
intenvoven" with a sprinkling of illustrative quotations. 
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Both of the preceding methods of compiling and reporting qualitative data were 
designed to address what until recently had been the biggest complaint about it: namely, 
that words were not as readily condensable as numbers. For all of the fearsome formulas 
and such, at least statistics (once eventually mastered and reasonably understood by their 
users!) enjoyed the reputation of being "concisely reportable." For instance, chi square and 
analysis of variance (ANOVA) tables are reasonably standard and focused in format. While 
words were readily acknowledged to be more revealing, easy to understand and 
undoubtedly more inherently interesting than numbers, they were thought to elude such 
condensation and concise compilation. That is, it was feared that the only way to report 
them would be to reproduce page after page of verbatim quotations - resulting in an 
oven\ helmingly lengthy report that no one would bother to r^ad from cover to cover. With 
the development of the matrix and summary narrative procedures, however, one could 
mdeed have the "best of both worlds." The researcher would enjcy the revelatory nature of 
the words as "packaged" in a more manageable fonnat, be it a shorter report or a creative 
graphic. 

This invariably leads to the idea: why not, indeed, employ the "best of both worlds" 
m ones research design? Can words and numbers be made to mix compatibly? Thanks to a 
very exciting new direction of research design, the answer is a resounding "yes." 

Having Ft All: .Multimethod Research Desio ns 

Multimethod research procedures are indeed the wave of the future in scholarly 
inquir> . due to their ability to integrate the precision of numbers with the rich, revelatory 
nature of the words. Brewer and Hunter ( 1989) have stated, "Its fundamental strategy is to 
aiiack a research problem with an arsenal of methods that have nonoverlapping weaknesses 
in udJitinn to complementary strengths (p. I7, emphasis in original text)." The use of more 
than one procedure to collect and analyze data furthermore tightens the internal validity or 
beiio>. ability of the study's Hndings and conclusions. Brewer and Hunter have explained 
this phenomenon of triangulation as follows: 

Broadly speaking, measurement is the operation of assignin<' either 
qualitative or quantitative values (that is, either names or numbers) to social 
phenomena. Triangulated measurement tries to pinpoint the value of a 
phenomenon more accurately by sighting in on it from different 
methodological viewpoints. To be useful, a measuring instrument mu-^t both 
give consistent results and measure the phenomenon that it purports to 
measure. When two reliable instruments vield conflicting results, then the 
validity of each is cast into doubt. When the findings of different methods 
agree, we are more confident (p. 17). 
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Packard and Dereshiu'sky (1992) have applied multimethodolosical research 
procedures to cross-validate data fr four alternative sources to obtain an in-depth 
understanding of the psychological environment of organizational settings and related 
human emotional responses. These three sources are: an intensive historical ("document 
analysis") I.terature review; two alternative forms of instrumentation (the Organizational 
Performance Scale (Packard, 1986) and the Perception Assessment Scale (Packard. 
Kundm and Biedein, 1986); and a series of focus group interviews conducted with 
graduate students enrolled in various research classes taught by the two authors. A matrix 
was developed which contains a comparative listing of the positive and negative factors of 
communication and associated feelings and emotional responses o: educational personnel 
(pgs. 5-6). These are grouped by: 1) Type and Level of Communication: 2) Receptor 
Feelmgs; and 3) Receptor Emotions. 

These findings, in turn, have been cross-validated through a factor analysis of the 
responses to the organizational climate items of the Perception Assessment Scale (Packard 
and Dereshiwsky, 1992). Four factors which accounted for 59.1% of response variance 
were identified and extracted. These were: 1) Program Reform and Accountabilitv 2) 
Commumcation and Emotional Health; 3) Evaluation and Placement; and 4) Psycholo'oical 
Self-Actualization. Individual sur^-ey items "clustering" with each factor (e.g., havina 
•robust loadings") were identified in this paper. 

There you have it: the statistician turned qualitative researcher who enjoys havina 
the best of both worids by using the two procedures effectively in research desions'! 
Having established the appropriateness of qualitative research methods, attention will now 
be turned back to "the numbers part" and how to make it work for YOU in on-site 
educational research desions. 



Making "Hi-Tech" \4Y Tprh- 
What You S hould Expect from a Good Tpchnical Sp eci^^li.t 

I admit it: I'm crazy about numbers. That's why I chose to major in applied 
statistics. In addition, I also love computers, having struggled to overcome an initial 
phobia which frankly developed in the days before the point-&-click Macintosh (Who 
wouldn't leam to freeze up at the thought of boxes of punched cards, hieroa|vphic-v 
programming code, and above all. reams of computer printouts with nothing butline afte'r 
ime of error messages that provided not a clue as to the one missing comma that actuallv 
tnggered ail 250of them?!) 
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Now. please look back at the word that precedes "statistics" in the second sentence 
of the paragraph that you just finished reading.' I want you to know that I take that word 
"applied" very, very seriously. For one thing, IVe always thought of myself as a teacher 
first and statistician second. Crunching away at a computer for twelve-hour days for a 
mega-insurance company in Hartford is certainly something that I'd be qualified to do: but 
frankly, it would hold absolutely no appeal for me. My joy comes from making statistics 
mean something real" to people. (I live for that visual light-buIb, that moment where IVe 
worked with someone on a quantitative procedure and I can see that they've grasped what 
the statistic "really means, in English.") 

To me, the real essence of statistics isn't in contorting the foot-long formulas: a 
computer can be programmed to do that for you. Rather, it is in matching the right tool to 
the nght need: understanding conceptually what a given statistic is supposed to do; being 
able to pick the right procedure(s) for an actual research question; and helping the originato^r 
of that question gain a solid understanding of what the statistic is doing - enough lo that 
he/she can see its purpose in the overall scheme of things. Thus, it may surprise you to 
know that probably 95% of my time with so-called "consulting clients" is spent, not in 
plugging away at computer keyboards or manipulating formulas, but rather in careful 
listening and talking about IDEAS. 

Furthermore, as mentioned in Barrier # 3, above, numbers may not necessarilv 
even be the best tool for you to use to address your research question. Ironically, for all of 
my love of things quantitative, since coming to Northern Arizona University I've spent the 
greater part of my time on qualitative data collection and analysis technique's. The research 
world is gradually coming to the (long-overdue) realizarion that words are superior to 
numbers in a great deal of context-rich research situations such as those that practitioners 
face. Interviews, log books, written diaries, both official and unofficial documents - these 
sources, and more, may be just what you need to answer your research question - instead 
of the computer programs and statistical tables. 

■'d like to think that I've acquired enough tools of the trade, both quantitative and 
qualitative, to be able to match the most appropriate procedure(s) to your research needs. 
The most rewarding part of my academic job assignment has been working with educators 
from a variety of diverse fields (e.g., educational leadership, curriculum and instruction: 
educational psychology; gifted and talented education) on the technical aspects of their 
research projects. Based upon my own experience, I'd like to offer you some tips for 
working effectively with technical specialists. These suggesrions are designed to make vour 
.nu.al venture into the worid of research not only (relatively) painless, but actu'allv 
rewarding, for you. 
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' • ^M}UQj^lc^ismjmM. Please don't think that you have to have an elaborately 
worked-out, jargon-heavy researchable question the first time you meet with a statistician, 
computer person or other technical expert. In fact, the sooner you can begin to bounce 
rough Ideas off, the better. You need to know if they're doable, testable, and the like - 
which may mean some give-and-take as you and the technician successively add. change, 
shape and refine your original idea. It's perfectly OK to start by saying, "I'm curious 
about X, but I don't have any idea how to turn this into a researchable topic/' That's a very' 
valuable starting point for further dialogue between you and your collaborator, because it 
says something about your general area(s) of interest. 

- Be patient with th e "fie l d-o uts i der" when exp lp^ining vour .ren nf ^nr.ro.r ifs a sad 
fact that the unknown (and therefore scary) is sometimes also given undue glory and 
importance. I've seen this phenomenon at work with respect to statistic'! ... and 
statisticians. 

Well ... I'd like to let you in on a little secret. When I meet you for the first time 
to find out about what you're interested in studying, I'm sitting there and wondering a bit 
anxiously, "I hope he/she will explain slowly enough and be patient with me, and not go 
TOO fast, so that I can follow what he/she is trying to do." In other words, your field 
jargon is as scary and intimidating to me as mine is to you! 

What this means is that I'll be needing to "do homework" - probably asking you 
lots of follow-up questions, and maybe even requesting additional readings and/or 
clanfication meetings - all designed to give me a good, thorough soaking in your field of 
study. This phase is akin to a second or third diagnostic medical opinion - time- 
consuming and perhaps tedious, yet essential to getting at the heart of reality. A "cood" 
technician will not rush to his/her computer before ensuring that he/she has acquired a^solid 
understanding of exactly what the researcher is trying to accomplish. Please be patient with 
us while we leam to "speak your language." 

Given the critical importance of the research question or problem, 1 should be able 
to restate ,t to you in my own words at the end of this phase. My practice is not to proceed 
with any analysis until I can say, "It seems to me that this is what you are tryina to find 
out. Have I restated it correctly?" If not, it is essential to keep the discussion g Jna until 
there is a meeting of the minds of both the researcher and the technician as to the^nTose of 
the research study. 

K now the research _roadma2. At this particular point I will tell my research 
collaborator(s), "It sounds to me like (Statistic X or Qualitative Procedure Y) would be best 
to answer your research question. But first, let me talk with you a little bit about what this 
procedure does and how it works." Granted, for practical purposes all parties mav have 
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agreed in advance that the technical specialist alone will be responsible for all of the 
computer programmirg and number-crunching aspects of the work. (This makes oood 
sense m terms of division of 'abor: after all, it's "what I do best!") However, this does 
NOT preclude the same honest two-way communication process that was applied to 
.dent.fymg a researchable problem. In essence, the roles are now reversed: the technical 
expert w.Ii be expected to do the same kind of patient explaining about the HOW as the 
practitioner did about the WHAT. 

As the source of the research problem or question, you are yourself a researcher 
worthy of respect and professional consideration. Do NOT. under any circumstances, 
allow for any of the following events to occur 

a) The technical expert simply "runs to his/her machine" without first telling you what 
he/she will be doing, and expects you to accept the end product(s) withou'i comment 
or question; 

b) The technical expert does tell you what procedure(s) he/she recommends, but 
"speaks in jargon" to you about them. You either dare not ask a question for fear of 
"looking dumb." or if you do. you are treated in a patronizing and/or dismissive 
manner ("What do you mean you didn't understand that?!"). 

Insist on your right to be an "informed consumer" and to have full and complete 
prior knowledge of the recommended ways to go about doing the research. There is one 
additional benefit to this step besides your basic right to know. For me, as the outside 
technical expert, it is one final "cross-check," if you will, that I did indeed understand the 
research question and what needs to be done to answer it. 

^""^ ample timf- for the data collection and an alysis pha.sP. . In my individual 
work with doctoral candidates on the analytic portions of their dissertations, I never cease 
to be amazed - and saddened - at how consistently they underestimate the amount of time 
.t will take to collect and analyze their data. For one thing, while computers have 
admittedly accelerated the time that it takes to create databases and perfomi calculations, it is 
NOT a matter of "pushing one button." as some people still seem to assume. In fact, some 
of the more sophisticated quantitative procedures (such as multiple regression ar ' 
categorical da:a analysis models) actually consist of a series of compuL runs and 
associated statistical tests. One needs to look at the outputs from step one to decide uhat 
vanables to include or exclude and repeat the process for step two. and so forth. 

Also, anyone who's ever had even the most basic computer programming course 
knows what a tedious and time-consuming process debugging a computer program can be. 
The best-laid plans" on coding sheets seldom, if ever, translate into a perfectly executed 
computer run on the first tr>'. Moreover, the causes of crashes can be as diverse'as a sin-le 
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comma or slash gone astray, to a typo in the database, to an internal system or hardware 
glitch that had nothing to do with your well-written program but caused it to dive-bomb 
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Qualitative data collection may initially look quite appealing as an alternative to such 
fearsome number-crunching. However, talk with anyone who has spent just 1 1/2 hours 
conducting a single focus group interview session and ask to see their notes and source 
tapes. The sheer volume of qualitative data is often an overwhelming surprise to those 
researchers who may have thought that it was "the easy way out" but v/ho must now spend 
hours and hours condensing these data into some sort of meaningful pattern. Furthermore, 
unlike the precision associated with inferential statistics, there simply is no "one right way" 
m condensing and summarizing qualitative data. The process is quite unlike arriving at a 
final t- or F-value which you can have a computer crunch out for you and which yo'u (or 
the machine) then look up against a value in a table to decide if it is "statistically 
significant." It is not unheard of, and in fact quite common, to decide that one's initial 
codmg or matrix breakdown scheme for summarizing qualitative data isn't working ... and 
to have to start ail over with a different model or prx)cedure. 

What this all boils down to is: be as patient with your technical expert during this 
stage as you were during the problem-definition stage (step #3). Do not put undue 
deadlmes or related pressures on your technical specialist to "crank out the results" without 
first consulting him or her as to what the projected time lines are likely to be. Or, better 
>et. ask your expert to describe to you the sequence of steps in data collection and analvsis. 
as well as a ballpark of how long (best- and worst-case scenarios) each step is likelv to 
take. I make sure that I have a thorough discussion with my client-collaborator(s) in which 
I spell out all of the key steps, from database formatting design through write-up of results. 
I also "gently warn" them of any potential stumbling blocks, such as not getting a 70^c 
survey response on the first mail-out, which could cause slowdowns, so that we can joint! v 
plan what to do if these contingencies should occur. That way, we all have a clear and 
agreed-upon idea ahead of time of what needs to be done and approximately how Ion 2 each 
step can be expected to take. 

5. Ma ke it vour own . Once the end product of analysis is completed, a "oood"" 
technical specialist will not simply "dump" it on you and expect you to instantiv understand 
.t. I actively work with my client-collaborators to explain the resulting statistical and/or 
qualitative (tabular or narrative) outputs. In the case of quantitative methods. I never just 
hand over a computer printout. Rather. I usually "distill" from it first, making shorter 
tables, graphs and charts in pencil or using a basic computer software graphics packase. 
Next. I sit down with my collaborator and go through these "more-bare-bones" resuhs 



taking care to explain each statistic or procedure thoroughly and, above all. Hnkinc it back 
to the research question so that the client can see what it was supposed to accomplish. 

As my introduction to research class knows ve^' well, a "good" researcher must 
never forget to "answer the question!" The goal of the technical specialist at this point is to 
help the chent-collaborator do exactly that. No one will ever expect you to "re-derive" your 
formula. They will, however, expect you to be able to tell someone what the statistic helped 
you to find out. and how that information can be used to answer your research question. 
Just as m step #3, insist on your right to know at this point. Do not accept jaroon, abstract 
formulas, and the like, as substitutes for a solid grounding as to what the research findinas 
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QlL^nd^nemore thing: statisticians are people, too! I know - this one surelv goes 
against the stereotype of your worst memories of the toughest statistics class you ever took. 
Horror stories about "inhuman" statistics and computer professors can be heard with 
alarming frequency within the walls of any graduate instituuon. (P.S. We all share in that 
nightmare: IVe had some terrible experiences along those lines myself.) As a result, it is 
tempting to think of statisticians as being in the same league with the machinery and 
formulas which we use to do our jobs - with not a single social skill in evidence. 

While I'm not denying that such stereotypes may have a grain of truth to them. I 
also beheve that it is dangerous to generalize from extreme cases. As I look back on mv 
own expenences. IVe also been most fortunate to work closely with researchers who are 
VERY MUCH "human" - who have taught me as much, if not more, about human 
development (and jovful living) than the mechanics of the research process. I also believe 
that It is their ver>' "humanity" that makes them such especially proficient technical experts 
and outstanding researchers, for they gain ready access to a wide variety of people places 
and situations as a result. As I indicated at the outset, I think of myself as a teacher first 
and a statistician or researcher second. I have also conscientiously tried to apply what my 
humanistic mentors have taught me in my own interactions with my students, clients and 
collaborators. In shon. I like to think that technical proficiency and humanity not onlv go 
hand in hand but work in synergy. 

There is tremendous potential for positive communication and interaction in ALL of 
us - regardless of who we are or what we do. Here are some thoughts on how to elicit 
them in the case of collaborating with a technical specialist: 

^' ^AUiiistjons if you're confused. Don't be afraid or ashamed to admit that vou 
don't understand something. (We ALL have to start somewhere.) 
Bvih^snme token, don't atrrihnt. "mind-readinc" cnpnhiljties to vnnr tPrhninnl 
exfiert. (We're not THAT talented!) If something confuses you. if you want to 
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make a change of direction, if you are frustrated, challenged, or just plain don't 
know where to go or what to do next ... speak up. Ambiguity is not only OK. but 
IS to be expected, as part of the research process. Communication does not consist 
of being a walking thesaurus; rather, it is a process which can be VERY non-linear. 
(A side note: linearity really exists only in analytic geometry and multiple regression 
classes - and even thess two have established procedures in case of departures 
from the deceptively simple and elegant straight line. Remember that.) 
Let the techni ca l e xpert a lso know when something is "very riohr " As 
implementers and teachers of traditionally difficult material, we statisticians and 
computer types are all too accustomed to "the negative:" confusion, frustration, 
anxiety, and the like. Well, having reminded you to KEEP telling us when these 
emotions are in fact present, I now feel compelled to focus on the other (perhaps 
rarer) extreme. It wouldn't hurt to let us know if we've done especially well at 
son.ething, too! That information, by the way, doesn't have to come in the form of 
effusive, gooey words of praise, either. It could be 2 direct, honest statement like. 
"I never really understood that statistic before, but from the way you just explained 
•t to me, I now have a feel for what it's supposed to do and how it works." (I've 
heard that, and it's just made my day.) 

LeL ys be a valued part of the p rorpss too . This point goes along with its immediate 
predecessor. While a computer can be turned on or off at will, I as a researcher and 
a person get mentally and actively involved with my topic. As a result, I inherently 
like to feel that I'm a valued member of the team. So ... don't let my involvement 
end with my deliver>' of the analytic results. Share your entire written draft with me 
and ask me for my reactions to the document as a whole. Tell me what you're 
going to do with the research results. If you're applying for a grant, keep me posted 
as to whether you got it. Invite me to come and listen, if possible, if you're 
presenting the results at some sort of official meeting. Or, better yet, offer me a 
chance at a co-authorship with you. including perhaps a joint presentation at a local, 
regional or national professional conference. Statisticians, practitioners, and 
content-area experts approach the research task from very different, but equallv 
valued, perspectives. There is no "better" or "worse" - ALL are equallv vital to it's 
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Good Help N'e ed Not Be THAT Hard tn RnH 
How to Locate YourTechnicnl 'fyp ^rt 



As with other professionals (e.g., physician, and lawyers), the same search 
maxims apply: shop around carefully and ask trusted sources fo^ refermls. An "out-of-the- 
phonc-book" approach may sound efficient but might well end in disaster. A "good Ht" 
mterpersonally is probably as important - if not more - than absolute credentials and 
technical expertise perse. 

For the educational practitioner, the first step might be to ask around within the 
dutnct to see if statistical consultants have recently been hired to assist with wide-scale 
research projects. He/she might also check with recent Ed.D. graduates who live and work 
nearby: colleges and universities with graduate programs often feature "help-for-hire" 
postings by computer and statistical specialists. Another prime source of names to call is 
the academic computing department of nearby universities. Employees may "moonlight" as 
consultants; or, if they are too busy themselves with university duties, they may in turn 
know of friends, recent graduates or other such contacts. 

As cautioned above and in the preceding section, make sure that YOU are 
comfortable with the technical expert BEFORE committing to a contract or other long-term 
busmess arrangement. Communication style, as well as statistical/computer substance, 
should be critical in your decision of working partner(s). Expertise must not masquerade as 
arrogance: a sincere respect for your respective position is absolutely essential, in mv 
opinion. 



Concluding Comments 

^ We've all no doubt heard the old maxim. "Anything worth doing is worth doing 
well." However, this saying does not automatically imply that we alone must do it aM - and 
do it Eerfectlv on our very Hrst tr>'. I am fond of telling my research students that""99^r of 
what I've learned on the computer. I've learned from having to undo catastrophic errors." 

I also share with my classes that "Good research is messy." The creative challense 
lies in trying to anticipate twists and turns in the road, as well as to have backup plans, 
procedures and resources ready to help at those particular points. 

As educational practitioners, you indeed have the worid at your disposal: that is. a 
world which is sometimes equally messy and occasionally leaves you with more problems 
than solutions. If you'll allow me yet .^.^her maxim at this point, you also know that 
smce "tomorrow is another day." you'll be back in the classroom facing that same concern. 

O . 1 q n, • 
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problem or challenge. This is precisely the point at which the research process stands 
ready to help you meet the challenges of addressing your needs to know. This readiness 
includes access to, and collaboration with, so-called "technical experts" in certain parts of 
the process, like me. The opportunities to change your world and mine, by each of us 
being who we are and applying the best of what we know as part of this process of 
discover}\ arc virtually unlimited. 
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